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Why GAO Did This Study 

The School Improvement Grant 
(SIG) program funds reforms In low 
performing schools. Congress 
provided $3.5 billion for SIG In fiscal 
year 2009, and a total of about $1 .6 
billion was appropriated In fiscal 
years 2010-2012. SIG requirements 
changed significantly In 2010. Many 
schools receiving SIG funds must 
now use the funding for specific 
Interventions, such as turning over 
certain school operations to an 
outside organization (contractor). 
GAO examined (1) what, If any, 
aspects of SIG pose challenges for 
successful Implementation; (2) how 
Education and state guidance and 
procedures for screening potential 
contractors and reviewing contractor 
performance compare with leading 
practices; and (3) to what extent 
Education’s technical assistance and 
oversight activities are effectively 
supporting SIG Implementation. GAO 
surveyed SIG directors In all 50 
states and the District of Columbia; 
analyzed Education and state 
documents; and Interviewed officials 
from 8 states and school districts In 
those states, SIG contractors, and 
education experts. 

What GAO Recommends 

GAO recommends that Education (1) 
provide additional support to states 
about making evidence-based grant 
renewal decisions and (2) ensure 
that contractor performance Is 
reviewed. Education generally 
supported our first recommendation 
but disagreed with the second. We 
modified our recommendation to 
address some of Education’s 
concerns. 

View GAO-12-373. To view an e-suppiement 
with more data see GAO-1 2-370SP. For more 
information, contact George Scott at (202) 
512-7215 or scottg@gao.gov. 



What GAO Found 

Successful SIG implementation posed a number of challenges. 
Specifically, state and district officials were challenged to build staff 
capacity and commitment for reform, facing difficulties such as recruiting 
and retaining strong staff members. In addition, the SIG requirements to 
develop teacher evaluations and increase student learning time were 
difficult to implement quickly and effectively because they required 
extensive planning and coordination. Furthermore, states sometimes had 
limited evidence about the performance of SIG schools when making 
grant renewal decisions. For example, although Education’s guidance 
identifies meeting annual student achievement goals as a key criterion for 
making renewal decisions, some states did not receive student 
achievement data by the time decisions had to be made. States also 
made decisions through qualitative assessments of schools’ 
implementation efforts, but such determinations were not always based 
on extensive interaction with schools or systematic monitoring. Education 
did not provide written guidance to states about making evidence-based 
grant renewal decisions after they encountered these challenges. 



Districts used a significant portion of their SIG funds to hire contractors for 
a range of services, such as managing school operations and conducting 
teacher professional development. Leading practices show that screening 
potential contractors and then reviewing their performance are important 
for ensuring accountability and quality of results. Education required 
screening of contractors before contract awards were made. However, 
Education did not require review of contractors during contract 
performance, and states varied in whether they ensured that contractors 
were reviewed during the course of contract performance. 

Education’s assistance and oversight activities are generally supporting 
SIG implementation. In our survey, nearly all states reported they were 
satisfied with Education’s technical assistance, particularly the agency’s 
SIG guidance and conferences. In addition, many states reported that 
Education’s guidance was timely. With respect to oversight. Education 
monitored 12 states in school year (SY) 2010-201 1 and found 
deficiencies in 1 1 of the 1 2 states. Education is working with states to 
correct these deficiencies. For SY 2011-2012, the agency plans to use a 
risk-based approach to conduct on-site monitoring in 14 additional states. 
To maximize its oversight resources. Education also plans to conduct 
some limited monitoring in five additional states in SY 201 1-2012. 
Education officials told us that they plan to monitor the remaining states in 
SY 2012-2013 and that these states represent a small percentage of SIG 
funds. 
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Vice Chairman 
Committee on Appropriations 
United States Senate 

The Honorable Tom Harkin 
Chairman 

The Honorable Richard C. Shelby 
Ranking Member 

Subcommittee on Labor, Health and Human Services, 

Education, and Related Agencies 
Committee on Appropriations 
United States Senate 

The School Improvement Grant (SIG) program is designed to fund 
significant reforms in low performing schools. Schools that receive SIG 
funding (SIG schools) receive up to $2 million annually for 3 years to 
improve student outcomes, such as standardized test scores and 
graduation rates. While the program — which provides funds to states on a 
formula basis — was first authorized in fiscal year 2002, it changed 
significantly in fiscal year 2009. Specifically, Congress greatly increased 
funding for the program from $125 million in fiscal year 2007 to $3.5 
billion in fiscal year 2009. From this amount, $3 billion was provided 
under the American Recovery and Reinvestment Act of 2009, and 
schools began spending these funds during school year (SY) 2010-2011.^ 
Additionally, a total of approximately $1 .6 billion was appropriated in fiscal 
years 2010-2012. 

In addition to funding increases, the U.S. Department of Education 
(Education) made major changes to SIG program requirements in 2010.^ 
As a result of these changes, many schools receiving SIG funding must 
now use the funding to implement one of four intervention models, each 
with specific requirements, such as replacing principals or turning over 
school management or certain school operations to an outside 



Vub. L. No. 111-5, 123 Stat. 115, 181. 
^75 Fed. Reg. 66,363 (Oct. 28, 2010). 
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organization (contractor). Also, states are now required to award SIG 
funds to school districts competitively, rather than by formula. States 
evaluate grant applications using several criteria, including the proposed 
intervention model for each school and the district’s budget and reform 
implementation plan, as well as their capacity and commitment to 
effectively implement the reforms. After grant awards are made, states 
are also responsible for deciding whether to renew schools’ SIG funding 
for additional years. State renewal decisions are to be based on criteria 
that can include whether school assessment results meet annual student 
achievement goals, and whether the school has implemented its chosen 
intervention model. Education oversees the program by reviewing state 
grant applications, monitoring implementation and providing technical 
assistance. 

The recent changes to the SIG program have raised questions about how 
grants are being implemented. The Senate Appropriations Committee 
and its Subcommittee on Labor, Health and Human Services, Education, 
and Related Agencies requested that we conduct a broad review of the 
SIG program, and in July 201 1 we reported on early implementation 
issues in six states during the first year of the expanded SIG program.^ 
That report found that the program’s implementation time frames in some 
cases did not allow schools sufficient time to plan and fully enact reforms, 
and we recommended that the Secretary of Education consider options to 
award SIG grants to school districts earlier in the school year. In this 
review, we addressed the following questions: (1) What, if any, aspects of 
SIG pose challenges for successful implementation? (2) How do 
Education and state guidance and procedures for screening potential 
contractors and reviewing contractor performance compare with leading 
practices? (3) To what extent are Education’s technical assistance and 
oversight activities effectively supporting SIG implementation? 

To identify aspects of SIG that pose challenges to successful school 
turnaround, we administered a survey to the 50 states and the District of 



^For more information, see GAO, School Improvement Grants: Early Implementation 
Under \Nay, but Reforms Affected by Short Time Frames, GAO-1 1 -741 (Washington, 
D.C., July 25, 2011). The six states were: Delaware, Nebraska, Nevada, Rhode Island, 
Ohio, and Virginia. 
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Columbia, and received a 100 percent response rate/ We also reviewed 
Education documents and interviewed Education officials. We gathered 
in-depth information from eight states that were selected to represent a 
range of size, geographic diversity, and the intervention models being 
used. These states were California, Delaware, Nebraska, Nevada, Ohio, 
Rhode Island, Texas, and Virginia. In each of the states, we also 
interviewed school district and school officials from one to three districts 
that had SIG schools. To collect information about policies and 
procedures for screening potential contractors and reviewing contractor 
performance, we examined Education guidance and state monitoring 
policies. We also interviewed Education officials and state and local 
officials from our site visit states. We identified leading practices for 
screening and reviewing contractors based on our past work. To evaluate 
Education’s technical assistance and oversight efforts, we reviewed 
relevant documents, such as SIG monitoring protocols, and interviewed 
relevant Education officials. We also gathered information from states 
through our survey and state site visits. We interviewed several 
Education-funded technical assistance providers that served our site visit 
states and various stakeholders, such as teachers’ union officials. For 
more information about our scope and methodology, see appendix I. 

We conducted this performance audit from January 201 1 to April 2012 in 
accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe that 
the evidence obtained provides a reasonable basis for our findings and 
conclusions based on our audit objectives. 



Background 



Education’s changes to SIG requirements in 2010 have led to new 
responsibilities for the agency, states, and school districts. These entities 
all play key roles in the SIG award and implementation process, with 
Education supporting and overseeing state SIG efforts. Before awarding 
formula grants to states. Education reviews each state’s application and 
approves the state’s proposed process for competitively awarding SIG 



'^This report does not contain all the results from the survey. The survey and a more 
complete tabulation of the results are Included in GAO-1 2-370SP. When discussing 
survey results in this report, the term “states” also Includes the District of Columbia. See 
appendix I for more information about our survey design and implementation. 
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grants and monitoring implementation. As part of the state application 
process, states identify and prioritize eligible schools into three tiers: 

. Tier I schools. Receive priority for SIG funding and are the state’s 
lowest-achieving 5 percent of Title I schools (or 5 lowest-achieving 
schools, whichever number is greater) in improvement status.® 

. Tier II schools. Secondary schools eligible for, but not receiving. Title I 
funds with equivalently poor performance as Tier I schools. 

• Tier III schools. Title I schools in improvement status that are not Tier 
I or Tier II schools. 



After states receive SIG funding, school districts submit applications to 
states describing their SIG reform plans for eligible schools. Education 
has required that districts base their plans on an analysis of each school’s 
needs, called a needs assessment. After reviewing district applications, 
states distribute their SIG dollars using their approved competitive grant 
award process, giving priority to districts seeking funding for Tier I and 
Tier II schools. Education’s regulations and guidance require districts 
receiving SIG awards for Tier I or Tier II schools to implement one of four 
intervention models in each school. Select aspects of each model are as 
follows: 

• Transformation. Transformation schools must replace the principal, 
implement a transparent and equitable teacher and principal 
evaluation system that incorporates student academic growth, identify 
and reward staff who are increasing student outcomes, and provide 
increased student learning time, among other requirements. 

• Turnaround. In addition to implementing many requirements of the 
transformation model, turnaround schools must use locally adopted 



®Title I, Part A of the Elementary and Secondary Education Act of 1965, as amended, 
provides funding to school districts and schools with high concentrations of students from 
low-income families. Under Title I, states set academic targets and measure schools’ 
progress in meeting them. Schools in improvement status have missed academic targets 
for at least two consecutive years. The definitions of Tier I and Tier II schools also include 
high schools that have a graduation rate of less than 60 percent over a number of years. 
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competencies to screen existing staff and rehire no more than 50 
percent of the existing staff. 

• Restart. The district must reopen the school under the management of 
a contractor, such as a charter school operator, charter management 
organization, or education management organization. 

• Closure. The district must close the school and enroll its students in a 
higher achieving school within a reasonable proximity. 



Districts may choose to use contractors to implement aspects of their 
reform plans. Schools enacting a restart model are required to contract 
with an organization that will assume many of the decision-making and 
leadership functions in that school. Districts employing other models may 
also contract with external organizations for services that could include 
data analysis, teacher professional development, and efforts to create 
safe school environments. Our work notes the importance of screening 
potential contractors before awarding contracts, as well as regular 
evaluation in order to ensure contractors are providing timely and quality 
services with government funds.® 

In addition to reviewing district applications, states are also responsible 
for monitoring grant implementation. States make decisions about 
whether to renew funding for each SIG school for an additional year, 
based on factors such as whether schools meet annual student 
achievement goals that districts set for the schools. Pursuant to 
Education’s guidance, if a school meets its annual goals, the state must 
renew the school’s SIG grant. If a school does not meet one or more 
annual goals. Education’s guidance gives states the flexibility to consider 
other factors such as the “fidelity with which the school is implementing” 
its chosen intervention model. ^ 

Education provides states with technical assistance and oversight 
regarding SIG implementation. For example. Education funds 21 
Comprehensive Centers that help build states’ capacity to assist school 



®GAO, Contract Management: Opportunities to improve Surveiilance on Department of 
Defense Service Contracts, GAO-05-274 (Washington, D.C.: Mar. 17, 2005). 

^We refer to this as fideiity of impiementation. Education’s guidance does not further 
define the process or factors states shouid use when making this determination. 
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districts and schools. Sixteen of these organizations serve states in 
designated regions, and 5 provide technical assistance on specific issues, 
such as teacher quality. In addition. Education funds Regional 
Educational Laboratories, a network of 10 laboratories that serve 
designated regions by providing access to applied education research 
and development projects, studies, and other related technical assistance 
activities. Education also monitors states’ implementation of SIG. This 
monitoring process consists of visits to selected states and several SIG 
districts and schools within the monitored states, followed by reports 
documenting any findings. States have an opportunity to respond to any 
findings before the release of Education’s monitoring reports. 

States have awarded funding to two cohorts of schools since the program 
was modified and expanded in 2010. In the first cohort, 867 schools 
received SIG funding to implement one of the four intervention models in 
SY 201 0-201 1 , and in the second cohort 488 schools received funding to 
implement one of the intervention models in SY 2011-2012. Seven 
states® have received waivers from Education to delay awarding funding 
to their second cohort of schools until SY 2012-2013 because of various 
issues, such as turnover of key staff in state educational agencies. The 
proportion of schools choosing each model was similar in both cohorts 
and, as shown in figure 1 , most schools chose to implement the 
transformation model. 



®The seven states are Alabama, California, Hawaii, Missouri, Rhode Island, Tennessee, 
and Vermont. In addition, the Bureau of Indian Affairs was granted a waiver. 
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Figure 1 : Percentage of SY 2010-2011 and 2011-2012 School Grantees Using Each 
Reform Model 



1 % 





] Closure 
] Restart 
Turnaround 
Transformation 
Source. GAO survey of states. 




Successful SIG 
Implementation Has 
Posed a Number of 
Challenges 



Staff Capacity and 
Commitment to Reform 
Posed Implementation 
Challenges 



Although most states have increased the amount of staff time devoted to 
SIG since the program was expanded, some states have struggled to 
develop the necessary staff capacity to successfully support and oversee 
SIG implementation because of budget constraints. In our survey, 29 
states told us that they have increased the staff time devoted to SIG since 
they first applied for SIG funds in the expanded SIG program. However, 
officials from four of the eight states we visited — California, Nebraska, 
Rhode Island, and Texas — told us that because of budgetary constraints. 
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the time staff could devote to administering the SIG program and 
monitoring district implementation was significantly limited. For example, 
officials in California said that as a result of the state budget crisis, the 
state legislature reduced the amount of SIG funds available for state 
administration from the allowable 5 percent® to 0.5 percent, limiting the 
number of staff available to administer the program and monitor districts. 
Several state officials we spoke with also reported that their existing 
workloads made it difficult to focus on SIG. For example, in several states 
the program was administered by officials who also had responsibilities 
for other major education programs, such as Race to the Top. 

In addition, state officials sometimes did not have expertise in supporting 
school turnaround efforts. Officials from Education and several states and 
research groups told us that SIG required states to support local reform 
efforts to a much greater extent than they had in the past, and staff in 
some states had not yet developed the knowledge base to fulfill these 
responsibilities. Even when states were able to develop expertise in 
school reform and hire necessary staff, officials from Education told us 
that personnel turnover in many states made it difficult to retain such 
knowledge. For example, Rhode Island officials said they encountered 
difficulties filling vacated positions because many nearby states were also 
recruiting from the same small pool of qualified applicants. Several states 
increased their capacity through actions such as contracting with 
nationally recognized experts to help them run their grant competitions, 
establishing school turnaround offices, or hiring turnaround specialists 
that regularly handled an individualized caseload of SIG schools. For 
example, 18 states created new turnaround offices to help districts 
implement SIG, according to our survey. 

In addition to these state level issues, many districts also struggled to 
develop the necessary staff capacity to implement successful school 
reforms. It was particularly difficult for schools to recruit and retain 
qualified staff members, according to many stakeholders, including 
officials from several states and districts we visited. They told us that SIG 



®Under federal law, states are allowed to reserve not more than 5 percent of SIG grant 
funds for administrative costs. 20 U.S.C. § 6303(g)(8). 

^°Race to the Top awards education reform grants to select states through a competitive 
grant process. Funding for the program was first authorized through the American 
Recovery and Reinvestment Act of 2009. 
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schools were sometimes in rural areas or needed staff to have expertise 
that was in short supply, such as experience with reform or specialized 
academic subjects. Among the 12 states that Education monitored during 
SY 201 0-201 1 , they found that 4 states did not ensure that turnaround 
schools met requirements to remove at least half of the schools’ staff and 
hire new staff for those positions based on staff effectiveness.^^ For 
example, Education found that one monitored district in Minnesota did not 
base hiring decisions on prospective teachers’ instructional effectiveness. 
In addition. Education found that three of the monitored states did not 
ensure appropriate replacement of the principal in turnaround or 
transformation schools. 

Moreover, some districts did not have staff with expertise in using 
performance and evaluation data — such as data on student 
performance — to inform plans for reforming schools and ongoing 
instructional improvements. Education officials said that, in many cases, 
school district staff were able to collect data, but did not have experience 
linking data to needed interventions. In addition, our review of the needs 
assessments districts were required to develop when planning SIG 
interventions showed that some were more extensive than others. Also, in 
one district we visited, the new teacher evaluation process did not include 
state assessment data on student achievement as one of the evaluation 
criteria, as required by Education. Several states, districts, and 
researchers identified promising practices for recruiting and retaining staff 
or improving data usage, such as developing “grow-your-own” leadership 
programs, conducting priority hiring for SIG schools, or hiring data 
coaches to help teachers collect and analyze student data. 

Districts also varied in their commitment to use SIG funds to enact major 
reforms. According to our survey, 35 of 51 states awarded grants to all or 
most Tier I applicants who applied for grants starting in SY 201 0-201 1 , 
but several officials from states we visited and research organizations 
reported that some districts receiving SIG grants were not prepared to 
make significant reforms. For example, officials in one large school district 
we visited told us they followed turnaround model requirements to rehire 
no more than 50 percent of teachers at a SIG school. However, the 
district officials said they relocated the released teachers to other SIG 



Using locally adopted competencies to measure the effectiveness ef staff, scheols 
implementing the turnaround model must screen all existing staff and rehIre no more than 
50 percent, and select new staff. 
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schools in their district because those schools had almost all of the 
vacancies. Similarly, in two states we visited, district officials moved a 
school’s previous principal into another leadership position on site so that 
person could continue to work in the school even after a new principal 
was assigned. State and district officials also cited instances where 
districts chose their SIG model for reasons other than its likelihood of 
improving student success. For example, the superintendent in one 
district told us they chose the restart model because they considered it 
less restrictive than other models. Although many states responding to 
our survey told us that all or most of their transformation model schools 
were operating very differently after the first year of SIG, 33 states said 
that at least some of these schools choosing the transformation model 
were not.^^ Figure 2 shows responses from these 33 states about 
whether inadequate action by SIG schools or districts was a reason the 
schools were not operating very differently. 



^^For each model, our survey asked whether all, most, around half, few, or none of the 
schools in a state were operating very differently after the first year of SIG Implementation. 
Thirty-three states told us that some (most, around half, or few) transformation schools 
were operating very differently after the first year, while 13 states told us that all 
transformation schools were operating very differently and 5 did not provide a response. 
For more information, about the extent to which schools were operating very differently 
after the first year of SIG and reasons cited by state officials, see our e-supplement GAO- 
12-370SP. 
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Figure 2: State Responses on Whether Inadequate Action by SIG Schools or 
Districts Was a Reason Schools Were Not Operating Very Differently 




Reason in none of schools 
Too early to tell or missing 
Reason in all schools 
Reason in around half of schools 
Reason in most schools 
Reason in few schools 



Source: GAO survey of states 



Key SIG Requirements Are 
Difficult to Implement, 
Especially Under Short 
Time Frames 



SIG requirements to increase student learning time and develop new 
teacher evaluation systems were particularly challenging for some 
districts to implement fully, especially given the significant time pressure 
that we identified as a concern in our previous report on SIG.^^ Indeed, 
Education found districts that were not appropriately implementing 



^^GAO-1 1 -741 . Both the turnaround model and the transformation model require school 
districts to provide increased learning time. Increased learning time is defined by 
Education’s guidance as significantly increasing the total number of school hours to 
include: (1) instruction in core academic subjects; (2) instruction in other subjects and 
enrichment activities that contribute to a well-rounded education; and (3) teachers to 
collaborate, plan, and engage in professional development. The transformation model also 
requires school districts to use “rigorous, transparent, and equitable evaluation systems 
for teachers and principals that take into account data on student growth as a significant 
factor.” 
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increased learning time requirements in about half of the states that it 
monitored during SY 2010-201 1 and in both states for which it had 
completed SY 2011-2012 monitoring reports by February 201 2. According 
to district officials, at least half the districts we spoke with will not have 
fully implemented new teacher evaluation systems by the end of their 
second year of SIG. In addition, during its SY 201 1 -201 2 monitoring visit 
to Iowa, Education found that student growth was not always incorporated 
in new teacher evaluation systems, as required. Our analysis showed that 
increased learning time and teacher evaluation requirements were 
challenging because the planning needed to implement them was 
complex and time-consuming, and stakeholders, such as unions and 
parents, were sometimes reluctant to embrace the changes. Some 
districts struggled to develop increased learning time initiatives that would 
be sustainable after their 3-year SIG grant ended. More specifically: 

• Interventions required extensive planning. Effectively implementing 
increased learning time and teacher evaluations required extensive 
planning. Several stakeholders stressed the importance of carefully 
designing increased learning time schedules because, for the 
intervention to be successful, it must provide quality instruction rather 
than simply increasing the amount of poor instruction. Officials from 
several districts said they were unable to fully implement their plans 
for increased learning time at the beginning of the first year of 
increased SIG funding because, for example, they first needed to fully 
analyze their existing schedules and curricula and adapt them to meet 
SIG requirements. Officials from states and districts we visited often 
stressed that developing a teacher evaluation system is time- 
consuming because it requires districts to accurately and 
comprehensively identify, collect, and analyze information about 
teachers’ performance and students’ academic growth. In response to 
the challenges involved in planning and implementing teacher 
evaluation systems. Education allowed states to apply for a waiver to 
extend the planning period for this requirement, and 27 states applied 
for and received the waiver as of February 2012. In its final SIG 
requirements. Education required schools implementing the 
transformation model to implement new teacher evaluation systems 
within the first year of the grant. Districts in states receiving these 
waivers must develop their evaluation systems during SY 2011-2012; 
pilot or fully implement them by SY 201 2-2013; and use them to make 
decisions about retention, promotion, and compensation by SY 2013- 
2014. The timeline is the same regardless of whether the SIG schools 
in the district are from the first or second SIG cohort. 
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• Stakeholders sometimes reluctant to embrace required changes. 
Implementation was also delayed or otherwise challenged by 
concerns from various stakeholder groups. Teachers and teachers’ 
unions were sometimes concerned about increasing student learning 
time or implementing new teacher evaluations in SIG schools, 
according to Education, state, and district officials. For example, these 
officials said unions were concerned about whether teacher 
evaluation systems that incorporated student academic growth could 
do so in a manner that would not penalize teachers working with the 
most challenging students. Such concerns sometimes led to delays in 
finalizing evaluation systems. In a few cases, officials told us that 
other stakeholder groups such as parents and school board members 
were also resistant to SIG requirements. For example, an official in 
Virginia said that some schools trying to increase learning time had 
met resistance from parents because students often had jobs or 
responsibilities at home once the traditional school day was over. 

. Difficulty designing sustainable approaches for increasing learning 
time. State and district officials also questioned whether increased 
learning time initiatives would be sustainable after SIG funds were 
exhausted. For example, survey respondents from 26 states said the 
costs of increased learning time were unlikely or very unlikely to be 
sustainable after the SIG grant ends, compared with 10 states that 
reported it was likely or very likely to be sustainable.^"^ Rhode Island 
officials noted that increased learning time benefits students enrolled 
in SIG schools during the grant cycle, but state and local financial 
constraints will make it difficult to sustain the increased learning time 
for future students. Due in part to these concerns, one of the two 
districts in the state with SIG schools limited the amount of learning 
time it added in order to avoid significant cuts in this time after grant 
funding ends. 

While many officials stressed the complexity of effectively implementing 
these requirements, some states and districts that we visited found ways 
to address the challenges they posed. This was particularly true in 
districts that had started to plan for and implement similar school reforms 
prior to applying for SIG funds. Many officials from Education, state, and 
districts stressed the importance of stakeholder involvement while 



^"^The remaining 15 respondents said that either it was as likeiy as uniikely that these 
costs wouid be sustainabie; they did not know; the question was not appiicabie, or it was 
too eariy to teil. 
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designing and implementing SIG reforms in order to enhance buy-in and 
strengthen reform initiatives. In order to increase students’ learning time 
without increasing teacher workloads and salaries, Education officials and 
researchers told us that a few districts were working with community 
partners to fund or staff additional learning time or were staggering 
teachers’ schedules so that students would be in class longer but 
teachers would not. In addition, a few states developed sample teacher 
evaluation system that met SIG requirements so that districts could use it 
as a framework for developing their own systems. 



Some States Had Limited 
Evidence about SIG Grant 
Performance When Making 
Grant Renewal Decisions 



States often had limited evidence for making decisions about whether to 
renew schools’ SIG funding. For example, in our survey, officials from 10 
states told us that they did not use schools’ achievement of annual goals 
to make grant renewal decisions after SY 2010-201 1 . According to state 
officials, at least half the states we interviewed did not have the annual 
student achievement data available at the time they had to make renewal 
decisions because assessment results only became available at the end 
of the summer. Officials from two of these states told us that timely 
access to annual achievement data will continue to be a problem in future 
years. In addition, even when these data were available, states frequently 
chose not to base their decisions on schools’ achievement of annual 
goals. Twenty-three of 44 states responding to our survey question said 
that, among schools that had their funding renewed, all or most did not 
meet their annual goals. Several officials from our site visits questioned 
the usefulness of annual goal data in determining whether progress was 
made, particularly because districts set their own performance targets. 

For example, California officials said they did not find annual goals data 
useful because districts often included generic annual goals in their 
applications for SIG funding instead of proposing goals based on schools’ 
unique circumstances. 



Regardless of whether annual goals information was available, states 
almost always considered “fidelity of implementation” — the extent to 
which the school is implementing the requirements of its intervention 
model — when making grant renewal decisions. However, states did not 
always base decisions about this criterion on extensive information. In our 
survey, 48 of 51 states identified fidelity of implementation as an 
important factor in their decision-making process, more than any other 
factor. Several states we spoke with said that qualitative information 
about implementation was important for assessing grant progress 
because the first steps of school reform, such as efforts to change school 
culture, do not always result in measurable student achievement gains. 
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However, making this assessment can involve a high degree of 
subjectivity and states’ determinations were not always developed based 
on extensive interaction with schools or systematic monitoring of their 
implementation efforts. For example, officials in California told us they 
used fidelity of implementation as their key criterion for making grant 
renewal decisions, and that the primary method for evaluating this 
criterion was one telephone conversation with each district at the end of 
the year. Prior to those conversations, the state had limited interaction 
with most districts for the purpose of assessing their implementation and 
was unable to conduct SIG monitoring visits for budgetary reasons. In 
addition, a Virginia official told us the state used fidelity of implementation 
for making renewal decisions but would benefit from guidance on how to 
define and measure it. 

States were in some cases reluctant to discontinue SIG funding even 
when information they collected showed that schools were not 
implementing key requirements with fidelity. Several officials from states 
we visited said they renewed all schools’ SIG funding even if the schools 
were struggling to fulfill key SIG requirements because tight 
implementation timeframes made the officials reluctant to eliminate 
funding after the first year of the grant. In our survey, 21 states reported 
that half or fewer of their Tier I and Tier II schools were able to implement 
major aspects of their plan by the beginning of SY 2010-201 1 , such as 
extending the school day or having new staff in place. In the 19 cases 
where these states had made renewal decisions, the state renewed all or 
most grants. Furthermore, officials in several states we visited identified 
instances where they chose to renew schools’ funding despite significant 
problems at the district or school level, such as having administrators who 
were not committed to enacting major reforms or were not ensuring that 
planned reforms were fully implemented. For example, officials from 
Nevada said they renewed such grants after the first year because they 
did not want to negatively impact students and teachers when significant 
district-level problems were outside their control. 

Although Education reviewed states’ proposed grant renewal procedures 
through the state SIG application process, the agency did not provide 
written guidance after grant renewal challenges arose. Education required 
states to submit their renewal processes for review as part of their SIG 
applications. Nonetheless, in several state applications we reviewed, 
descriptions of renewal processes and criteria did not align with the 
practices the state actually implemented. For example, states that told us 
they were unable to use annual goals data to make renewal decisions 
had originally identified these goals as a key renewal criterion in their 
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applications to Education. In its work with states, Education officials told 
us they found some had difficulty using annual goals data or fidelity of 
implementation and that the agency provided technical assistance to 
several states that asked for help. However, agency officials were not 
aware of how states ultimately addressed these issues, and said the 
agency has not provided any additional technical assistance on grant 
renewal.^® 

States renewed almost all SIG grants at the end of SY 2010-201 1 , and in 
some cases imposed conditions on schools for renewal. According to 
Education, 39 states chose to renew funding to every SIG school in their 
state. Eleven states and the District of Columbia chose not to renew 
funding to one or more schools, for a total of 16 nonrenewed schools 
overall.^® Of these 16 schools, about two thirds were not renewed due to 
problems with fidelity of implementation. Several states we spoke with 
chose to renew grants with conditions or required changes. For example, 
officials in Ohio told us that struggling schools were required to take 
corrective actions in the second year of the 3-year grant and that their 
level of success in taking such actions will be a key criterion in future 
renewal decisions. In addition. New York officials renewed all grants after 
SY 2010-201 1 under the condition that transformation and restart schools 
would implement state and federal SIG teacher evaluation requirements 
by December 30, 201 1 . Once that deadline passed, state officials 
determined that no districts had met the requirements and suspended all 
SIG funding until they were met. In February 2012, the state 
commissioner reinstated funds to half of the SIG districts after 
determining that the districts had made the necessary changes. In our 
survey, 23 states reported that at least a few of their SIG schools were 
required to make major changes to their SIG plans as a condition of 
having funding renewed. 



^®Education officials told us they will review state renewal processes through monitoring 
starting in SY 201 1 -201 2, but that their SY 201 0-201 1 monitoring took place prior to states 
making renewal decisions. 

^®The Bureau of Indian Affairs also chose not to renew two schools. 
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States and Districts 
Screen Contractors 
Before Selecting 
Them to Receive SIG 
Funding, but 
Inconsistent 
Approach to 
Reviewing 

Performance Reduces 

Contractor 

Accountability 



Contractors Provide a 
Wide Range of Services, 
and Many Receive 
Significant Amounts of SIG 
Funding 



Contractors provide a wide range of services with SIG funds, and school 
districts have often given contractors major roles in schools using the 
restart, turnaround, and transformation models. Education’s guidance 
identifies a clear role for contractors in schools using the restart model. 
Specifically, districts must hire a contractor to take over school 
operations. For example, in the Los Angeles Unified School District, the 
Partnership for Los Angeles Schools has been given full management 
authority over five restart schools. In contrast. Education allows districts 
with schools using the turnaround and transformation models — which 
include more than 90 percent of schools receiving SIG funds — to use 
contractors, but does not identify a specific role for them. Most turnaround 
and transformation schools we visited were working with contractors. 
Although in some cases turnaround and transformation schools used 
these contractors for minor tasks, in other cases the contractors played a 
major role in school operations. For example, in Virginia, the state 
required schools implementing the turnaround and transformation 
intervention models to use a contractor for a range of services that could 
include improving teacher performance, principal and management 
leadership, or changing school culture. 



Among the school districts we visited, several planned to spend 
significant amounts of their SIG grants on hiring contractors. These 
included districts using the restart, turnaround, and transformation 
models. For example, a district with one SIG school using the 
transformation model planned to spend about $450,000 for contractors in 
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one school year. In addition, a district that we visited with three SIG 
schools planned to spend approximately $1 .5 million on contractors over 
the 3-year period for services that included data analysis and curriculum 
planning. 

Our prior work and reports regarding services acquisition have shown the 
importance of building safeguards into acquisition processes to ensure 
accountability. These leading practices include screening potential 
contractors prior to award using a thorough selection process that 
evaluates their ability to achieve results and the contractors’ past 
performance.^^ Once a contractor has been selected, officials should 
routinely review contractors’ work to help ensure they are providing timely 
and quality services and to help mitigate any contractor performance 
problems.^® 



Education and States 
Required Contractors to 
Be Screened, but 
Screening Procedures 
Varied 



Education required and states reported requiring that potential contractors 
be selected after a thorough screening process. Education required that 
either states or districts screen contractors prior to contract award to 
ensure their quality.^® Although Education’s guidance does not provide 
specific criteria for approval. Education requires each state to describe in 
its state application how it will ensure that school districts screen 
contractors. Each of the eight states we reviewed required districts to 
describe their plans to screen contractors in their applications for SIG 
funding. 

In addition, states varied in how they approached contractor screening at 
the state level, either taking an active role in the process or delegating 
screening responsibilities to districts. According to our survey, 17 of the 



^^GAO, Federal Contractors: Better Performance Information Needed to Support Agency 
Contract Award Decisions. GAO-09-374, (Washington, D.C.: Apr. 23, 2009). 

^®For the purposes of this report, we define review during the course of contract 
performance as monitoring or quality assurance surveillance, as described in GAO-05- 
274. 

^^Education’s guidance requires that contractors implementing the restart model must be 
selected through a “rigorous review process,” which permits a district to examine a 
prospective contractor’s plans and strategies, and to ensure that the contractor has the 
capacity to implement its strategies. The guidance also requires districts to screen all 
other contractors for quality, stating that this screening serves a similar purpose as the 
rigorous review process used for restart contractors. 
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51 developed approved lists of contractors from which districts could 
choose. For example, Virginia officials told us they enacted statewide 
contracts with four organizations, and strongly encouraged districts to 
choose one of those four organizations. Ohio officials said they developed 
a list of approximately 100 state-screened organizations from which 
districts could choose, but districts were free to use other contractors, 
provided that they screened those organizations. States that we visited 
that did not develop a list of approved contractors reported requiring 
districts to screen contractors. For example, Texas officials told us they 
required all districts to use a formal competitive process in selecting 
contractors, which included a process to evaluate contractor proposals, in 
order to be approved by the state. 



Education Does Not 
Require that All 
Contractors’ Performance 
Be Reviewed, and State 
Review Approaches Vary 



Education’s monitoring protocols for the SIG program require the review 
of contractors in schools using the restart model, but they do not require 
review of contractors during contract performance for the other school 
improvement models. Education’s protocol for monitoring states’ SIG 
implementation asks whether districts have included accountability 
measures in the contract for restart schools and also asks for the district’s 
current assessment of the contractor. The protocol does not include a 
similar question for turnaround and transformation schools.^® 



States varied in their approaches to the review of contractors, and in 
some cases reported that they did not require that districts review 
contractors during contract performance. Among the eight states we 
spoke with, none assessed districts’ plans to review contractors in their 
SIG applications. In addition, several states reported not having any 
state-level review requirements. For example, Nebraska state officials 
said their districts conduct informal reviews of the contractors, but the 
state does not require reviews or provide districts with a formal process or 
metrics to assess performance. Similarly, in follow up calls for our state 
survey, state officials in several states said they do not require districts to 
review contractor performance and were unaware of whether districts 



^°Education officials said they included this requirement for restart schools because their 
regulation requires districts to contract with external management organizations and that 
the contractors are selected through a rigorous review process. Education officials also 
said that they include review requirements for restart contractors because those 
contractors are an essential part of the restart model, and that they have more flexibility 
than the contractors used in a school implementing a different model. 
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conducted any reviews. In contrast, Nevada officials told us they require 
districts to add accountability steps for contractors in each phase of work. 

Inconsistent review of contractors during contract performance reduces 
states’ and districts’ ability to ensure that they are receiving the services 
they have paid for. In our work, one stakeholder told us that in the 
absence of stronger guidance or oversight, the extent to which contracts 
include accountability measures is largely dependent on the knowledge 
and experience of the individual contract manager. Although some district 
officials in our site visits described efforts to include accountability 
measures or regular review in the contracts, others indicated that 
contractors are reviewed informally, if at all. 
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Education’s 
Assistance and 
Oversight Are 
Supporting SIG 
Implementation 



Education’s Technical Education’s guidance and technical assistance on SIG implementation 

Assistance Was Well was well received by nearly all states. In our survey, nearly all states 

Received by States responded favorably about Education’s guidance and various technical 

assistance offerings for SY 2011-2012. Most states reported that 
Education’s guidance and technical assistance were helpful and many 
reported they were very helpful (see fig. 3). In our survey, we also 
inquired about the amount and timeliness of guidance provided by 
Education. Forty-one states reported that Education provided about the 
right amount of guidance for the second year of SIG. In addition, 33 
states responded that in SY 2011-2012, Education’s guidance was timely, 
allowing the state to meet its needs, while 14 states commented that the 
guidance was not timely. 
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Figure 3: Survey Results Regarding the Helpfulness of Education’s Technical 
Assistance Offerings 

Technical assistance offerings 
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Note: The data for this graphic is from our survey, in which we asked states about the heipfuiness of 
specific technicai assistance offerings, in addition to the answers presented, we aiso inciuded “do not 
know” as a potential response. We also asked states about the usefulness of Education’s 
communities of practice, the implementation initiative, and the on-site monitoring visit. For these three 
items, the number of states responding “do not know” was 23, 30, and 30, respectively. Because the 
“do not know” responses were numerous, we did not include them in our graphic. 



Although most states told us that Education’s guidance was helpful, some 
identified additional technical assistance that would assist with SIG 
implementation. In an open-ended question on our survey that asked 
about the types of additional guidance that Education could provide, 15 
states indicated they wanted additional information about other states’ 

SIG implementation efforts that are working well. Several states that we 
met with also mentioned wanting more information on successful and 
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sustainable implementation strategies, proven contractors, increased 
learning time strategies, and teacher/principal evaluation systems. 

To provide additional support and enhance information sharing among 
the states. Education has recently begun three new assistance efforts. 
First, Education selected nine states to participate in the SIG 
“implementation support initiative” as an optional technical assistance 
resource.^^ Under this initiative, each participating state receives a visit 
from an Education representative as well as officials from the eight other 
participating states. These site visits have two purposes — first, to provide 
technical assistance to the states, and second, to enable states to 
engage in peer-to-peer information sharing. Education reported that they 
have used information from these site visits to produce targeted technical 
assistance reports. Second, in December 2011, Education began 
conducting monthly check-in calls with state officials to better manage 
SIG implementation. Each state was assigned an Education program 
officer responsible for providing oversight and technical assistance 
support, including outreach and monthly check-in calls. Lastly, Education 
launched the School Turnaround Learning Community — an on-line forum 
to provide states and districts with access to resources and to facilitate 
networking. According to Education, this initiative offers research-based 
practices and practical examples from states, districts and schools for 
developing and implementing SIG. 



Education is Now Using a 
Risk-Based Approach to 
Monitor All States within 3 
Years 



Education’s oversight strategy is to monitor all states during the 3-year 
period^^ — starting with SY 2010-201 1 — in which the first cohort of schools 
will receive SIG funding. In selecting states for on-site monitoring for SY 
2010-201 1 , Education did not use a SIG-specific risk-based approach 
and instead used the existing Title I monitoring schedule. However, due 
to resource constraints. Education suspended its Title I monitoring and 



According to Education officials, 17 states volunteered for this initiative, and the 9 states 
selected to participate were Alabama, Arizona, Arkansas, Connecticut, Louisiana, 
Michigan, Oklahoma, South Carolina, and Virginia. States that had received an Education 
SIG monitoring visit or were awarded a Race to the Top grant were not selected to 
participate. 

^^Education officials also told us that nine of these states received technical assistance 
visits through the implementation initiative. 

^^Education monitors how states receiving Title I grants Implement and administer the 
Title I program to determine whether states comply with program requirements. 



Page 23 



GAO-12-373 School Improvement Grants 





instead focused exclusively on SIG monitoring. Education also delayed 
SY 2010-201 1 monitoring to allow states and districts time to implement 
SIG before beginning monitoring in February 201 1 . For SY 2010-201 1 , 
Education conducted on-site monitoring in 12 states^"^ uncovering 28 
deficiencies. At least one deficiency was identified in 1 1 of the 1 2 
monitored states, with California and Pennsylvania having the most 
deficiencies with seven and five, respectively. Half of the monitored states 
had deficiencies in ensuring appropriate district implementation of the 
increased learning time requirement. In addition, two states did not 
ensure that all SIG funds were used consistent with the SIG 
requirements.^® 

In SY 2011-2012, Education selected states with a risk based approach 
tailored for SIG based on factors such as the size of a state’s SIG grant. 
For SY 2011-2012, Education officials initially selected 12 states to 
conduct on-site monitoring. As of February 2012, Education had issued 
SY 2011-2012 monitoring reports for Iowa and Florida, containing seven 
and two deficiencies respectively. For example, in Iowa, Education found 
that funds were not used consistently with SIG grant requirements nor 
was the state monitoring SIG as written in its approved SIG application. 
Education also set aside a portion of its oversight resources so that 
additional states could be selected for monitoring as more information 
became available. As of February 2012, Colorado and South Carolina 
were also selected to receive an on-site review. To maximize its oversight 
resources. Education plans to conduct some limited “desk monitoring” in 



SY 201 0-201 1 , Education conducted monitoring site-visits to the foiiowing states: 
Caiifornia, Indiana, Maine, Michigan, Minnesota, Mississippi, Montana, Nebraska, 

Nevada, Pennsyivania, South Dakota, and Tennessee. Education officiais toid us that they 
aiso conducted piiot monitoring site-visits in Maryiand and New Mexico, but these visits 
were iimited and used to test monitoring questions 

^®Education required states to take corrective actions to address identified deficiencies. 
Education officiais commented that states must submit evidence to prove that the 
identified deficiencies have been addressed. Education wouid then evaiuate the strength 
of the evidence to determine if it adequateiy addressed the weakness. Education began 
issuing its monitoring reports in the spring of 201 1 and as of March 2012, seven states — 
indiana, Maine, Michigan, Mississippi, Nevada, Pennsyivania, and South Dakota — had 
adequateiy addressed ail of the weaknesses identified from their monitoring report. In 
addition, two states, Minnesota and Nebraska, have addressed some of the identified 
weaknesses, but stiil have other weaknesses to resoive. 

^®For SY 2011-2012, Education initialiy pianned to conduct monitoring site visits to the 
foiiowing 12 states: Fiorida, Georgia, Hawaii, iliinois, iowa, Missouri, New Mexico, New 
York, Oregon, Rhode isiand, Texas, and Wisconsin. 
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five additional states in SY 2011 -201 2. According to Education, the desk 
monitoring protocol is similar to the on-site visit protocol, but — unlike the 
on-site monitoring — does not include interviews with school officials. 
Finally, Education officials told us that they plan to monitor the remaining 
states in SY 2012-2013, and that these states represent a small 
percentage of SIG funds. 



Conclusion 



Dramatic funding increases in a short period of time — such as those 
made to SIG — can subject federal programs to considerable financial risk. 
While states and school districts carry a large share of the responsibility 
for planning and implementing successful SIG reforms. Education also 
plays a critical role in supporting these efforts and mitigating risk through 
strong oversight and accountability. For example, it is important that 
Education have rigorous processes for reviewing state SIG applications, 
conducting oversight, and providing technical assistance when needed. 
The ability to successfully carry out these functions is vital to ensuring the 
long-term success of the SIG program and protecting taxpayer funds from 
waste and abuse. Although SIG has been challenging to implement, in 
part due to the short implementation timeframes we highlighted in our 
July 201 1 report. Education has reviewed state SIG applications, 
distributed funds to states, begun its monitoring activities, and provided 
technical assistance. However, the agency’s guidance in some cases has 
not been sufficient to ensure that schools and contractors are fully 
accountable. For example, given the implementation issues we and 
Education’s monitoring have found, it is critical that states have rigorous 
SIG grant renewal procedures in place to identify schools that are not 
making progress. Education has provided limited guidance to states 
about how to make renewal decisions. Some states are using highly 
subjective review processes to renew nearly all grants, often without key 
information on SIG schools’ performance. Until Education provides 
additional support about how states should make evidence-based 
renewal decisions when, for example, state assessment results are 
received too late to be factored into these decisions, schools that are not 
making progress may continue receiving SIG funds. In addition, although 
contractors are receiving large amounts of many schools’ SIG funds. 
Education has not ensured that states or districts review contractor 



of February 2012, Education had identified Idaho and Wyoming as states that wouid 
receive desk monitoring. Education has compieted the desk monitoring for idaho. 
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performance during the terms of their contracts. Unless Education takes 
action to ensure that states or districts review contractor performance, 
districts may not receive an appropriate level of contractor services for 
their SIG funds and funds may not be well spent. 



RGCOmiriGndationS for renewal decisions serve to hold districts and 

schools accountable, we recommend that the Secretary of Education 
ExGCUtivG Action provide additional support to states about how to make evidence-based 

grant renewal decisions, particularly when states do not have annual 
student achievement goal information available at the time renewal 
decisions are made. 

To ensure that contractors hired with SIG funding are accountable for 
their performance, we recommend that the Secretary of Education take 
steps to ensure that the performance of SIG funded contractors, including 
those in turnaround and transformation schools, is reviewed during 
contract performance. In developing such requirements and to ensure 
that those reviews are targeted to contractors receiving large amounts of 
SIG funding. Education could consider setting a dollar threshold amount 
for contracts, above which contractor performance should be reviewed. 



AgGncy ComiriGnts 
and Our Evaluation 



We provided a draft copy of this report to the Department of Education for 
review and comment. Education’s comments are reproduced in appendix 
II. Education generally supported our recommendation about SIG grant 
renewal and outlined how the agency is planning to address this 
recommendation. 



Education did not agree with our draft recommendation that it should 
require states to ensure that the performance of all SIG-funded 
contractors be reviewed, including contractors in turnaround and 
transformation schools. In its comments. Education said that it believed 
that existing provisions and requirements address this issue 
appropriately. For example. Education cited a federal regulation that 
requires districts to follow their existing procurement procedures, and 
noted that districts and states have their own requirements for evaluating 
contractors to ensure accountability. Education also said that the type of 
evaluation process needed for a contractor should depend on the 
contractor’s role, and that contractors used by schools implementing the 
turnaround or transformation models may be working on small, discrete 
projects and may require less provider-specific reviews than contractors 
in schools implementing the restart model. 
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We agree with Education that the need for performance reviews should 
be dependent on the specific role of the contractor, and we modified our 
recommendation to address some of Education’s concerns. Specifically, 
Education may wish to create a dollar threshold above which 
performance reviews are required. We continue to believe, however, that 
the current monitoring framework is inadequate. As noted in our report, 
schools implementing the turnaround and transformation models account 
for the overwhelming majority of SIG schools, and contractors operating 
in these schools are performing a range of functions, including some that 
are large or complex. In our view, there is a need for additional steps to 
ensure adequate review of contractor performance. Furthermore, our 
work shows that as a practical matter, states varied in their approaches to 
contractor review, with some imposing no requirements on districts. 
Education says that it will clarify in existing guidance the requirement for 
SIG recipients to follow state and local procurement procedures. 
Education could use this opportunity to implement our recommendation 
through additional guidance on contractor performance reviews. 

In addition. Education implied that our report was based only on the first 
year of SIG implementation. This is inaccurate. We also conducted 
interviews with all eight states, reviewed SIG documents, received 
finalized survey responses, and interviewed Education officials several 
times during the second year of implementation, thereby enabling us to 
reflect activities beyond the first year. Based on the number and 
significance of deficiencies identified in Education’s SIG monitoring 
reports — including some completed during the SY 2011-2012 — as well as 
our own findings, we continue to believe that Education should take 
additional steps to increase program accountability. 

Education also provided technical comments that we have incorporated 
into the report as appropriate. 
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We are sending copies of this report to relevant congressional 
committees, the Secretary of Education, and other interested parties. In 
addition, this report will be available at no charge on GAO’s website at 
http://www.gao.gov. 

If you or your staff have any questions about this report, please contact 
me at (202) 512-7215 or scottg@gao.gov. Contact points for our Offices 
of Congressional Relations and Public Affairs may be found on the last 
page of this report. GAO staff who made key contributions to this report 
are listed in appendix III. 



George A. Scott, Director 

Education, Workforce, and Income Security Issues 
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Appendix I: Objectives, Scope, and 
Methodology 



This study’s objectives were to answer the following questions: (1) What, 
if any, aspects of the School Improvement Grant (SIG) program pose 
challenges to successful implementation? (2) How do U.S. Department of 
Education (Education) and state guidance and procedures for screening 
potential contractors and reviewing contractor performance compare with 
leading practices? (3) To what extent are Education’s oversight and 
technical assistance activities effectively supporting SIG implementation? 
To meet these objectives, we used a variety of methods, including 
document reviews of Education and state documents; a web-based 
survey of the 50 states and the District of Columbia; interviews with 
Education officials and stakeholders; site visits to and teleconferences 
with 8 states; and a review of the relevant federal laws, regulations, and 
guidance. The survey we used was reviewed by Education and several 
external reviewers, and we incorporated their comments as appropriate. 

We conducted this performance audit from January 201 1 through April 
2012 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit to 
obtain sufficient, appropriate evidence to provide a reasonable basis for 
our findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our findings 
and conclusions based on our audit objectives. 



Providing Information 
on Challenges to 
Successful SIG 
Implementation 



To identify aspects of the SIG program that pose challenges for 
successful SIG implementation, we analyzed responses to our survey of 
state educational agency officials with responsibility for SIG in the 50 
states and the District of Columbia. The wed-based survey was in the 
field from August to October 201 1 . In the survey, we asked states to 
provide information on challenges they faced in implementing the SIG 
program and on other aspects of the program, such as SIG grant 
renewal. We received responses from all 50 states and the District of 
Columbia, for a 1 00 percent response rate. We reviewed state responses 
and followed up by telephone and e-mail with select states for additional 
clarification and context. 



Nonsampling error could affect data quality. Nonsampling error includes 
variations in how respondents interpret questions, respondents’ 
willingness to offer accurate responses, and data collection and 
processing errors. We included steps in developing the survey, and 
collecting and analyzing survey data to minimize such nonsampling error. 
In developing the web survey, we pretested draft versions of the 
instrument with state officials in various states to check the clarity of the 
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Appendix I: Objectives, Scope, and 
Methodoiogy 



questions and the flow and layout of the survey. Education officials also 
reviewed the draft survey and provided comments. On the basis of the 
pretests and reviews, we made minor revisions of the survey. Using a 
web-based survey also helped remove error in our data collection effort. 
By allowing state SIG directors to enter their responses directly into an 
electronic instrument, this method automatically created a record for each 
SIG director in a data file and eliminated the errors (and costs) associated 
with a manual data entry process. In addition, the program used to 
analyze the survey data was independently verified to ensure the 
accuracy of this work. Detailed survey results are available at 
GAO-1 2-370SP. 

We also conducted site visits to and teleconferences with eight states — 
California, Delaware, Nebraska, Nevada, Ohio, Rhode Island, Texas, and 
Virginia — that reflect a range of population size, number of SIG schools, 
and use of the four SIG intervention models. In each state, we 
interviewed state officials, as well as district or school officials from one to 
three districts that had Tier I or Tier II SIG schools. Districts were selected 
in consultation with state officials to cover heavily and sparsely populated 
areas, and a variety of SIG intervention models. We also reviewed 
documents, such as state and district applications for SIG funds, and the 
relevant federal laws, regulations and guidance.^ We interviewed 
Education officials and stakeholders, such as teachers’ union officials 
from the national and local levels. 



Identifying How 
Education and State 
Guidance on 
Contractors Compare 
with Leading 
Practices 



To gather information about state policies and procedures for selecting 
and overseeing contractors, we analyzed state survey results. Our survey 
questions included whether states had developed a list of approved 
contractors, the SIG turnaround models for which they required that 
districts work with contractors, and whether states reviewed contractor 
performance. We reviewed Education documents, including SIG 
guidance, the state application template, and monitoring protocols, and 
interviewed Education officials responsible for reviewing state 
applications and providing oversight of states. Further, we reviewed state 
and district SIG applications from the eight states to identify their 
selection and review processes for contractors, and proposed contract 



^This includes Department of Education, Office of Inspector General, School Improvement 
Grants: Selected States Generally Awarded Funds Only to Eligible Schools, ED- 
OIG/A05L002 (Washington D.C., March 2012). 
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expenditures. We also spoke with state and local officials about their 
procedures for selecting and overseeing contractors, as well as with 
several contractors working with districts we visited. We compared 
Education and state requirements for selecting and overseeing SIG 
contractors to leading contracting practices that were identified through 
collaboration with our contracting experts and review of GAO-09-374^ and 
GAO-05-274.3 



Describing the Extent 
to Which Education 
Has Supported SIG 
Implementation 



To address the extent of Education’s support and oversight of SIG 
implementation, we reviewed Education guidance, summaries of 
Education assistance, monitoring time frames, monitoring protocols, and 
monitoring reports from SY 2010-2011. In addition, we analyzed survey 
results. We asked states to provide information on the federal role in SIG, 
including their perspectives on technical assistance offered by Education 
and Education’s monitoring process. We also talked with officials from 
Comprehensive Centers and Regional Educational Laboratories serving 
several of the eight states we worked with. The technical assistance 
providers were selected to include those working with large, medium, and 
small rural states. In addition, we interviewed Education officials in charge 
of the Comprehensive Centers Program and in charge of SIG monitoring 
efforts. 



^GAO-09-374. 

^GAO-05-274. 
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United States Department of Education 

WASHINCilON. D.C. 20202 



March 23,2012 
Mr. George Scott 

Director, Education, Workforce, and Income Security Issues 
U.S. Government Accountability Office 
441 G Street, NW 
Washington, DC 20548 

Dear Mr. Scott; 

Thank you for the opportunity to comment on the Government Accountability Office’s 
(GAO’s) draft report, “School Improvement Grants: Education Should Take Additional 
Steps to Enhance Accountability for States, Schools and Contractors” (12-373). We 
appreciate the time that your office has committed to this study and your efforts to gather 
grantee perspectives on implementation. 

The School Improvement Grants (SIG) program focuses on turning around the country ’s 
persistently lowest-achieving schools. It requires States and districts to target rigorous 
interventions, and funding to support those interventions, at those schools that have 
continued to fail students year after year. Given the difficulty and complexity of some of 
these very important tasks, we anticipated that States and districts would encounter various 
challenges, and we developed extensive guidance and provided technical assistance to help 
them meet those challenges. While we agree that tackling specific aspects of SIG 
requirements may not be easy for some districts and schools, it is nonetheless important 
work that is already showing great promise and early positive results. 

States and districts have demonstrated courageous leadership in implementing reforms that 
require them to do things in a different way than they have done before. And districts and 
their school leaders and teachers across the country are taking on their duties under SIG 
responsibly and effectively. We are encouraged that most of your survey respondents 
reported that most or all of their schools implementing the transformation model are 
operating very differently after the first year of SIG. That means that hundreds of the 
lowest-performing schools across the country are making dramatic changes under the SIG 
program. While initially we were not expecting significant increases in achievement gains 
after the first year of implementation, there are preliminary data from 43 States that show 
meaningful improvement 

To provide some further background on the program, SIG seeks to accelerate achievement 
in our nation’s lowest-performing five percent of schools through far-reaching 

V%WW,ed.gOV 

Ai)0 MARYLAND AVH., SW, WASHINGTON, DC 2U202 

71je Deparment of Education ‘s mission is to promote student ochievemem and preparation for global competitiveness by 
fostering educational excellence and ensuring mval access. 
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interventions. Federal grants support school leaders, teachers, unions, and local partners in 
the community to undertake this challenging work. We are seeing some very encouraging 
signs of progress in the first year of data from SIG schools, which few experts anticipated. 

At the national level, the No Child Left Behind law required persistently low-performing 
schools to take action to improve student learning. But in reality, chronically under- 
performing schools were required to do little — and for years the Federal government 
provided little or no money to help support change in these schools. Almost no high 
schools, for example, were included in the earlier Federal school improvement efforts — 
even though just 1 5 percent of our nation’s high schools produce half of the nation’s 1 .2 
million dropouts. 

SIG has impacted over 1 ,200 schools, each of which received a three-year grant of up to 
$2 million a school. In the first cohort of schools, 45 percent were high schools. Some of 
the nation’s high schools present some of the toughest challenges. We are still getting in 
the results of the first year of the SIG program. But preliminarj' data show that after Just 
one year, the commitment to change is producing dramatic gains in learning in a 
significant number of schools. 

There are about 850 schools in the first SIG cohort. We now have preliminary 
achievement data from 43 States, covering about 700 of those schools in their first year of 
the program. In year one, roughly one in four schools saw double-digit increases in 
mathematics proficiency. About one in five schools had double-digit increases in reading 
proficiency. In roughly 60 percent of SIG schools, the percentage of students who were 
proficient in mathematics or reading went up in the first year of the program. 

As encouraging as these increases in academic achievement are, they are still preliminary'. 
We will need several years of data to confirm a lasting improvement in academic 
achievement. We are also continuing to gather data on other critical outcomes that matter 
to assessing student progress, such as changes in graduation rates, dropout rates, discipline, 
attendance, and other indices. 

States are taking very seriously their responsibility to award grants and provide effective 
oversight and support to their districts and schools. We are encouraged that, according to 
your survey, 29 States increased the amount of staff dedicated to the SIG program since 
2009 and 50 States linked each SIG school or district with a designated State support 
person or team. Moreover, 3 1 States have visited SIG districts at least once a month for 
monitoring or technical assistance, and all have been providing technical assistance to their 
districts through webinars, conferences, and telephone calls. 

We also agree that States have some varying levels of capacity to implement the SIG 
program, especially because they have been required to provide new levels of support and 
oversight for their lowest-performing schools. For this reason, the Department has focused 
on providing technical assistance to States to help build capacity. In addition to our 
guidance and application support, we launched the online School Turnaround Learning 
Community, whieh now has over 3,000 members, to provide webinars and tools to support 
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implementation of challenging interventions like increased learning time and teacher 
evaluations. We also initiated a peer-to-peer technical assistance initiative that brings State 
SIG leads to other States to observe and leant from promising practices in those other 
States. Wc arc pleased that, as the draft report noted, the Department’s technical assistance 
has been well received by nearly all the States: 44 States reported that the Department’s 
guidance and resources arc helpful or very helpful and all 1 1 States in the peer-to-peer 
initiative found it helpful or very helpful. 

In the fall of 201 1, the Department also created the new Office of School Turnaround 
(OST) in the Office of Elementary and Secondary Education to administer the SIG 
program. OST has focused on increasing foraial monitoring, as well as regular 
communications with SIG grantees through frequent check-in calls. As a result, the 
Department has more infonnation to address implementation challenges and identify 
opportunities to target technical assistance and oversight to where they are most needed. 
Because we understand that States want even more opportunities to learn from each other, 
we continue to provide those peer-to-peer efforts though additional conferences, the online 
School Turnaround Learning Community, and a second round of the peer-to-peer technical 
assistance initiative. 

Our responses to your specific recommendations for the Department are as follows: 

Recommendation #1: To ensure that SIG grant renewal decisions serve to hold 
districts and schools accountable, we recommend that the Secretary of Education 
provide additional support to states about how to make evidence-based grant renewal 
decisions, particularly when states do not have annual student achievement goal 
information available at the time renewal decisions are made. 

We have no significant objection to Recommendation 1 , since the Department is already 
providing such additional support to help States with these issues. It should be noted, 
however, that two different types of evidence may inform a State educational agency’s 
(SEA’s) evidence-based decision regarding whether to renew a local educational agency’s 
(LEA’s) SIG grant with respect to a particular school or schools; ( 1 ) “results-based” 
evidence, or evidence related to achievement and progress on the SIG leading indicators; 
and (2) “process-based” evidence, or evidence related to how a school or LEA is 
implementing a SIG intervention model and whether that implementation is in accordance 
with the lea’s approved application and the requirements of the SIG program. 

With respect to “results-based” determinations, the Department recognizes that turning 
around the nation’s lowest-achieving schools, which by definition have experienced year 
after year of failure, is challenging work that does not always bring immediate results. The 
Department also recognizes that improvement in academic achievement often may not be 
evident in one academic year, even though we have seen some early successes in the SIG 
program. With this in mind, the Department issued final requirements for the SIG program 
that require SEAs to renew an LEA’s SIG grant with respect to all schools that are meeting 
their achievement goals and making progress on the SIG leading indicators, and that 
peimit SEAs to renew the grants even with respect to schools that are not making their 
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goals. The SIG requirements also pennit SEAs to set their own criteria for making 
renewal decisions with respect to grants for schools that are not meeting their achievement 
goals and making progress on the leading indicators. 

Consistent with the Department’s view about the time it may take to improve achievement 
in persistently lowest-achieving schools, many SEAs were reluctant to temiinate SIG 
grants at the end of the first year of implementation for a school’s failure to meet its 
achievement goals and to demonstrate progress on the leading indicators, as the SEAs felt 
there had not been sufficient time for schools to demonstrate progress in improving student 
achievement. Additionally, as noted in the GAO draft report, student achievement data 
were not available in many States at the time renewal decisions were made at the end of 
the first year, but became available during the second year of implementation. Given State 
timelines in reporting achievement data, we agree that SEAs would benefit from some 
additional guidance on how to make “results-based” renewal decisions when they do not 
have annual achievement data available at the time renewal decisions are made. 

With respect to “process-based” determinations, under section 440 of the General 
Education Provisions Act, 20 U.S.C. § 1232c, an SEA has authority to take certain 
enforcement action, including suspending or withholding payments under a Federal 
program, if an LEA has failed to comply substantially with the Federal requirements 
applicable to that program. OST has confirmed that, consistent with this authority, at least 
eight SEAs made “process-based” determinations to decline to renew SIG grants for 
twelve schools in nine LEAs because the schools were not implementing their SIG 
interventions with fidelity to the SIG final requirements. 

OST maintains regular communication with States regarding renewal decisions through 
monthly check-in calls, reviews of SIG applications, and on-site and desk monitoring. 

SEAs were not yet making renewal decisions during the first monitoring cycle. However, 
OST is monitoring renewal decisions during the second monitoring cycle. OST will 
examine the criteria used by States that have declined to renew one or more SIG awards to 
determine whether such criteria were appropriate, and will provide additional guidance 
and/or technical assistance to all States based on those findings. Additionally, OST will 
update guidance to clarify that SEAs may terminate a grant if a district fails to comply with 
program requirements, even in the absence of data on annual goals, and provide guidance 
on using other data, such as data on SIG leading indicators, as part of renewal decisions. 
Finally, the requirement that fiscal year (FY) 201 1 applicants provide a signed assurance 
that they will use the renewal process identified in the FY 20 1 0 application to determine 
whether to renew an LEA's grant illustrates the importance of the renewal process. 

Recommendation #2: To ensure that contractors hired with SIG funding are 
accountable for their performance, we recommend the Secretary of Education 
require states to ensure that the performance of all SIG funded contractors, including 
those in turnaround and transformation schools, is reviewed during contract 
performance. 
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For the reasons outlined below, we do not agree with this recommendation. This issue is 
already addressed appropriately and adequately through existing provisions and 
requirements, and we will continue to monitor these matters and provide further assistance 
when appropriate to drive continuous improvement. The Department requires that LEAs 
screen external providers to ensure their quality and has emphasized the importance of this 
screening process through guidance, a Dear Colleague Letter, and recurring technical 
assistance efforts. SEAs and LEAs are also required to monitor implementation of the SIG 
grants and school progress, which includes the performance of SIG contractors. The 
Department believes, however, that the type of evaluation process necessary for 
contractors paid with SIG funds depends on the contractors’ role in supporting SIG 
models. For example, the Department has required that LEAs hold external providers 
under the restart model accountable for meeting the SIG requirements because they are 
engaged in whole-school reform and have considerable flexibility with respect to the 
school improvement activities they will undertake. External providers supporting the 
implementation of a turnaround or transformation model may require less provider-specific 
monitoring, as they generally work on specific, discrete activities to improve student 
academic achievement. 

Further, under 34 C.F.R. § 80.36(a), when procuring services under a SIG grant, an LEA 
must follow the same policies and procedures it uses for procurements from non-Federal 
funds. As part of their policies and procedures for procurements from non-Federal funds, 
districts and States have their own requirements and practices for screening and evaluating 
external partners and for ensuring appropriate controls and accountability in this context. 
While we continue to review and monitor this area carefully, if is not always necessary to 
establish new Federal requirements on top of this type of State and local requirements. 

The Department does not believe it is necessary or appropriate to review State procurement 
laws as part of the SIG application process, but it is important for States and districts to 
follow these laws. OST will clarify through our guidance that States and districts are 
required, w'hen using SIG funds, to follow their State and local procurement laws and 
evaluate external providers where required under those laws. 

Overall, we believe that in a number of aspects, the draft report presents an incomplete and 
somewhat misleading view of how well SIG is being implemented. It is important to note 
that the draft report examines the first year of a redesigned program that dramatically alters 
how States and districts intervene in their lowest-performing schools. While there is 
always room for some improvements in the early stages of implementation, and the 
Department is committed to continuous improvement in implementing its programs such 
as SIG, the draft report presents some information that should be modified and clarified, 
including the title of the report. As noted in the enclosed technical comments, we believe a 
more appropriate title would be “Education Should Provide Some Additional Guidance 
and Assistance to Further Improve School Improvement Grants Implementation by States, 
Schools and Contractors.” Please see the enclosure for a more detailed discussion of this 
issue and other suggested edits to the report. 
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Thank you again for the opportunity to comment on the draft report. Please Set us know if 
you have any questions or would like to discuss our comments and suggestions. We look 
forward to receiving the final report. 



Sincerely, 




Michael Yudin 
Acting Assistant Secretary 
for Elementary and Secondary Education 

Enclosure 
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